Information processing apparatus, information processing method, and program

ABSTRACT

An information processing apparatus includes a controller (100) that performs: control of displaying information in augmented reality in a preparation mode in which a real object is disposed, the information that supports disposition of the real object on the basis of prerecorded information about the real object with which a virtual object has an interaction; and control of recording disposition result information of the real object in a disposition result information database referred to in a normal mode in which the virtual object that has the interaction with the disposed real object is displayed.

TECHNICAL FIELD

The present disclosure relates to an information processing apparatus,an information processing method, and a program.

BACKGROUND ART

In recent years, there is a technique to provide a user with informationby superimposing and displaying a virtual object on a real object.

For example, PTL 1 described below discloses a technique to recognize anobject (a real object) in a moving image captured from an ever-changingpoint of view and maintain a state in which associated information ofthe object is displayed.

CITATION LIST Patent Literature

PTL 1: Japanese Unexamined Patent Application Publication No. 2015-95147

SUMMARY OF THE INVENTION

In a case where an imaginary object is superimposed and displayed on areal object, it is possible to produce an interaction between the realobject and the virtual object (for example, such as that the virtualobject collides with the real object and a sound is produced, or thatthe virtual object sits on the real object); however, to provide suchuser experience, it is desirable to dispose a designated real object ina designated place in advance. If a virtual object performs apredetermined interaction despite the fact that a designated real objecthas not been disposed in a designated place, the interaction looks likean unnatural action, which causes a feeling of strangeness.

Accordingly, the present disclosure proposes an information processingapparatus, an information processing method, and a program that make itpossible to support disposition of a real object related to a virtualobject and prevent deterioration of user experience in a case where thevirtual object is displayed.

According to the present disclosure, there is proposed an informationprocessing apparatus including a controller that performs: control ofdisplaying information in augmented reality in a preparation mode inwhich a real object is disposed, the information that supportsdisposition of the real object on the basis of prerecorded informationabout the real object with which a virtual object has an interaction;and control of recording disposition result information of the realobject in a disposition result information database referred to in anormal mode in which the virtual object that has the interaction withthe disposed real object is displayed.

According to the present disclosure, there is proposed an informationprocessing method that is performed by a processor, the informationprocessing method including: displaying information in augmented realityin a preparation mode in which a real object is disposed, theinformation that supports disposition of the real object on the basis ofprerecorded information about the real object with which a virtualobject has an interaction; and recording disposition result informationof the real object in a disposition result information database referredto in a normal mode in which the virtual object that has the interactionwith the disposed real object is displayed.

According to the present disclosure, there is proposed a program causinga computer to function as a controller that performs: control ofdisplaying information in augmented reality in a preparation mode inwhich a real object is disposed, the information that supportsdisposition of the real object on the basis of prerecorded informationabout the real object with which a virtual object has an interaction;and control of recording disposition result information of the realobject in a disposition result information database referred to in anormal mode in which the virtual object that has the interaction withthe disposed real object is displayed.

As described above, according to the present disclosure, it is possibleto support disposition of a real object related to a virtual object andprevent deterioration of user experience in a case where the virtualobject is displayed.

It is to be noted that above-described effects are not necessarilylimitative; in addition to or in place of the above effects, there maybe achieved any of the effects described in the present specification orother effects that may be grasped from the present specification.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is an explanatory diagram of an overview of an informationprocessing system according to an embodiment of the present disclosure.

FIG. 2 is a diagram illustrating an example of an entire configurationof the information processing system according to the presentembodiment.

FIG. 3 is a block diagram illustrating an example of a configuration ofan information processing terminal according to the present embodiment.

FIG. 4 is a diagram illustrating an example of real object informationaccording to the present embodiment.

FIG. 5 is a flowchart illustrating an example of the flow of operationprocessing in a preparation mode according to the present embodiment.

FIG. 6 is a flowchart illustrating an example of the flow of operationprocessing in a normal mode according to the present embodiment.

FIG. 7 is an explanatory diagram of alternative processing in a casewhere a real object has not been disposed according to the presentembodiment.

MODES FOR CARRYING OUT THE INVENTION

Hereinafter, description is given in detail of preferred embodiments ofthe present disclosure with reference to the accompanying drawings. Itis to be noted that, in the present specification and drawings, repeateddescription is omitted for components substantially having the samefunctional configuration by assigning the same reference numerals.

Furthermore, description is given in the following order.

1. Overview of Information Processing System According to Embodiment ofPresent Disclosure 2. Configuration of Information Processing Apparatus3. Operation Processing

3-1. Operation Processing in Preparation Mode

3-2. Operation Processing in Normal Mode

4. Conclusion Summary 1. OVERVIEW OF INFORMATION PROCESSING SYSTEMACCORDING TO EMBODIMENT OF PRESENT DISCLOSURE

FIG. 1 is an explanatory diagram of an overview of an informationprocessing system according to an embodiment of the present disclosure.An information processing terminal 10 according to the presentembodiment displays, in augmented reality, information that promptsdisposition of a real object with which a virtual object has aninteraction, thereby supporting the disposition of the real objectrelated to the virtual object and preventing deterioration of userexperience when the virtual object is displayed.

As described above, in a case where an interaction is produced between areal object and a virtual object (for example, such as that the virtualobject collides with the real object and a sound is produced, or thatthe virtual object sits on the real object), if the designated realobject has not been disposed in a designated place in advance, a stateof the virtual object becomes unnatural, which causes deterioration ofuser experience (for example, such as that a collision sound is producedout of nowhere, or that the displayed virtual object looks unnaturallylike floating).

Furthermore, in a case of an event application performed at a particularplace, an operating side has to dispose a real object necessary for theevent application; however, it is difficult to dispose the designatedreal object in a position and direction that a developer intends andcheck if the disposition state is correct.

Accordingly, in the embodiment of the present disclosure, there isprovided a preparation mode in which information that promptsdisposition of a real object is displayed in augmented reality on thebasis of information about the real object with which a virtual objecthas an interaction, which makes it possible to more clearly indicate thedisposition (a position and direction) of the real object to a user(hereinafter, also referred to as a “setter”) who prepares the realobject.

Specifically, for example, as illustrated in FIG. 1, in a case where itis necessary to dispose a real object (for example, a garbage can) in adesignated position in a real space, the information processing terminal10 displays, in augmented reality, an image 42 (a virtual object) of thereal object in a designated disposition place. The image 42 of the realobject may be a 3D model image for the preparation mode, or may havetransparency.

This enables the user to intuitively grasp the disposition place and thedirection of the real object, and makes it possible to prevent a feelingof strangeness in a case where the disposition and the direction of thereal object are different at the time of execution of an application.Furthermore, by the image 42 of the real object being displayed, theuser is able to intuitively grasp the size of the real object as well.

It is to be noted that the information about the real object with whichthe virtual object has the interaction may be embedded in theapplication in advance. The information about the real object (such as ageneral name, a shape, material information, and dispositioninformation) is inputted by the developer in implementing behavior (theinteraction) of the virtual object, and is embedded in the applicationin association with the virtual object (for example, together with an IDof the virtual object). In the present specification, the applicationmeans software including a program that controls augmented realitydisplay of a virtual object, information about a real object to bedisposed, display data of the virtual object, sound data at the time ofan interaction, etc. The information about the real object is notlimited to be embedded in the application; alternatively, a database (areal object information database) may be generated separately, forexample, in a server 20 or somewhere and be referred to through theinformation processing terminal 10 as appropriate.

Furthermore, the information processing terminal 10 may further displayan image 44 that indicates detailed information about the real object tobe disposed (for example, a name, a shape, a material, a size, a color,points of concern about disposition, etc.) as illustrated in FIG. 1.

Moreover, as illustrated in FIG. 1, the information processing terminal10 may display, as a test, a virtual object 40 that has an interactionwith the real object to be disposed. This makes it possible to graspwhere and what interaction is actually produced and dispose the realobject more accurately. Furthermore, in a case where a substitute forthe designated real object is disposed, checking an interaction of avirtual object allows the substitute to be disposed not to becomeunnatural.

According to the present embodiment, even in a case where a developerand an executor (an operator) of an application are different, it ispossible to easily set an environment in which the application is to beexecuted. Even in a case where a real object to be used is changed withfrequent update of an application, it is possible to easily handle thecase by referring to real object information embedded in the updatedapplication or updated real object information DB.

Next, the information processing terminal 10 acquires disposition resultinformation of the real object. For example, upon completion of thedisposition of the real object, the user (the setter) performs anoperation (such as a gesture operation or a voice input) to check off acheckbox 441 displayed in the image 44. In a case where the checkbox 441has been checked off, the information processing terminal 10 recordsinformation about the disposition of the real object having beencompleted as disposition result information. In a case where the userhas not prepared the designated real object, and has disposed asubstitute, the material, the size, etc., of the substitute may beinputted. Furthermore, in a case where the user has unavoidablydisplaced the disposition place or the direction, how much thedisplacement has been is inputted. Moreover, in a case where the userhas not disposed the real object, having not disposing the real objectis inputted.

The disposition result information is referred to at the time ofexecution of the application, more specifically, in a normal mode inwhich the virtual object is displayed. Upon displaying the virtualobject in an interaction point, the information processing terminal 10is able to grasp the disposition state of the real object by referringto the disposition result information. This enables an appropriateinteraction to be performed not to become unnatural, and makes itpossible to prevent deterioration of user experience.

Specifically, the information processing terminal 10 performs apredetermined interaction in a case where the designated real object hasbeen disposed, and, in a case where there is a change, the informationprocessing terminal 10 performs alternative processing in accordancewith the change. For example, the information processing terminal 10changes the collision sound in accordance with the material of thesubstitute (for example, the collision sound is normally a metallicsound; however, in a case where the material of the real object ischanged to a material that is not metal, the collision sound is changedin accordance with the changed material), adjusts the display positionof the virtual object in accordance with the size of the substitute, ordoes not produce an interaction in a case where no real object has beendisposed, which makes it possible to prevent the interaction of thevirtual object from becoming unnatural.

As above, the information processing system according to the embodimentof the present disclosure has been described. It is to be noted that amethod for the augmented reality display in the present embodiment isnot particularly limited. For example, an eyeglass type HMD (HeadMounted Display) as illustrated in FIG. 1 may be used as the informationprocessing terminal 10. The eyeglass type HMD is provided with a displaysection 120 in a lens part located in front of user's eyes when the userwears the eyeglass type HMD. The display section 120 is realized by anoptically transmissive display, and the user is able to visuallyrecognize an imaginary object while viewing a real world with a nakedeye at the same time. Furthermore, the HMD is not limited to a HMD thatpresents an image to both eyes, and may be a HMD that presents an imageonly to one eye. Moreover, as illustrated in FIG. 1, the HMD is providedwith an outward-facing camera 111 that captures an image in a user'sgaze direction, i.e., the user's field of vision when the user wears theHMD. Furthermore, the HMD may be provided with a microphone(hereinafter, referred to as a “mike”) and a speaker.

Moreover, the HMD is not limited to the eyeglass type, and may be aheadband type (a type that is worn with a band running around the entireperimeter of the head, and may sometimes be provided with a band runningnot only the side but also the top of the head) or a helmet type (that avisor portion of a helmet corresponds to a display).

Furthermore, the information processing terminal 10 according to thepresent embodiment may be a video see-through HMD. In a case of a videosee-through HMD, the display section 120 located in front of the user'seyes when the user wears the video see-through HMD is realized by avideo transmissive display. The video transmissive display has astructure in which a field of vision is covered with a display, andallows a real world and an imaginary object to be viewed as an imagecaptured in real time.

Subsequently, an entire configuration of such an information processingsystem according to the present embodiment is described with referenceto FIG. 2. FIG. 2 is a diagram illustrating an example of the entireconfiguration of the information processing system according to thepresent embodiment.

As illustrated in FIG. 2, the information processing system according tothe present embodiment includes the information processing terminal 10and the server 20. The information processing terminal 10 and the server20 are coupled through a network 30, and are able to transmit andreceive data.

As described above, the information processing terminal 10 is a displaydevice that includes the display section 120 and is able to display avirtual object in augmented reality. An application that performsdisplay of the virtual object may be received from, for example, theserver 20. A specific configuration of the information processingterminal 10 will be described later with reference to FIG. 3.

The server 20 is a content server, and transmits, for example, theapplication that performs display of the virtual object to theinformation processing terminal 10. Furthermore, the server 20 mayaccumulate disposition result information of a real object and transmitthe disposition result information to the information processingterminal 10 as necessary.

As above, the system configuration example of the information processingsystem according to the present embodiment has been described. It is tobe noted that the above-described configuration described using FIG. 2is just an example, and the configuration of the information processingsystem according to the present embodiment is not limited to thisexample. For example, functions that the information processing terminal10 according to the present embodiment and the server 20 have may berealized by a single device. The configuration of the informationprocessing system according to the present embodiment is flexiblymodifiable in accordance with the specifications and operation.

2. CONFIGURATION OF INFORMATION PROCESSING TERMINAL 10

FIG. 3 is a block diagram illustrating an example of the configurationof the information processing terminal 10 according to the presentembodiment. As illustrated in FIG. 3, the information processingterminal 10 includes a sensor section 110, a controller 100, the displaysection 120, a speaker 130, a communication section 140, an operationinput section 150, and a storage section 160.

<2-1. Sensor Section 110>

The sensor section 110 has a function of acquiring (sensing) variousinformation about a user or surroundings. The controller 100 of theinformation processing terminal 10 is able to perform head tracking,position tracking, spatial recognition, gesture recognition, voicerecognition, etc. on the basis of the various information sensed by thesensor section 110.

For example, the sensor section 110 may include the outward-facingcamera 111, an inward-facing camera 112, a mike 113, a gyro sensor 114,an acceleration sensor 115, a direction sensor 116, and a positionmeasurement section 117. It is to be noted that the specific example ofthe sensor section 110 mentioned here is an example, and the presentembodiment is not limited thereto. For example, the sensor section 110may include an illuminance sensor, a biosensor, etc. Furthermore, eachof the sensors may include a plurality of sensors.

The outward-facing camera 111 and the inward-facing camera 112 eachinclude a lens system including an imaging lens, a diaphragm, a zoomlens, a focus lens, etc., a drive system that causes the lens system toperform focusing and zooming, a solid-state image sensor array thatphotoelectrically converts imaging light obtained by the lens system andgenerates an imaging signal, etc. The solid-state image sensor array maybe realized by, for example, a CCD (Charge Coupled Device) sensor arrayor a CMOS (Complementary Metal Oxide Semiconductor) sensor array.

It is desirable that the outward-facing camera 111 be set to an angle ofview and a direction that include at least an area corresponding to theuser's field of vision. Furthermore, a plurality of the outward-facingcameras 111 may be provided. Moreover, the outward-facing camera 111 mayinclude a depth camera that is able to acquire a depth map by sensing.

The mike 113 collects the user's voice and an environment sound of itssurroundings, and outputs them as voice data to the controller 100.

The gyro sensor 114 is realized by, for example, a 3-axis gyro sensor,and detects an angular velocity (a rotation speed).

The acceleration sensor 115 is realized by, for example, a 3-axisacceleration sensor, and detects an acceleration at the time ofmovement.

The direction sensor 116 is realized by, for example, a 3-axisgeomagnetism sensor (a compass), and detects an absolute direction (anorientation).

The position measurement section 117 calculates a self-location of theinformation processing terminal 10. For example, the positionmeasurement section 117 may use so-called SLAM (simultaneouslocalization and mapping) as a method for self-localization.

Furthermore, the position measurement section 117 may detect the currentposition of the information processing terminal 10 on the basis of asignal acquired from the outside. Specifically, for example, theposition measurement section 117 may be realized by a GPS (GlobalPositioning System) positioning section, and receive a radio wave from aGPS satellite and detect a location in which the information processingterminal 10 exists. Moreover, the position measurement section 117 maydetect the location by means of, for example, Wi-Fi (registeredtrademark), Bluetooth (registered trademark), transmission and receptionwith a mobile phone, a PHS, a smartphone, or the like, or short-rangecommunication or the like.

<2-2. Controller 100>

The controller 100 functions as an arithmetic processing device or acontrol device, and controls all operations in the informationprocessing terminal 10 in accordance with various programs. Thecontroller 100 is realized by an electronic circuit, for example, suchas a CPU (Central Processing Unit) or a microprocessor. Furthermore, thecontroller 100 may include a ROM (Read Only Memory) that stores aprogram, a calculation parameter, etc. that are used and a RAM (RandomAccess Memory) that temporarily stores a parameter, etc. that change asappropriate.

The controller 100 performs head tracking, position tracking, spatialrecognition, gesture recognition, voice recognition, etc., and maydisplay a virtual object appropriately in accordance with anapplication. More specifically, the controller 100 according to thepresent embodiment functions as a user situation recognition section101, a surroundings situation recognition section 102, a dispositioninformation management section 103, a mode setting section 104, and adisplay control section 105.

(User Situation Recognition Section 101)

The user situation recognition section 101 recognizes, as a usersituation, a position (for example, a position in a room, aself-location), a posture, an action, etc. of the user on the basis ofvarious sensing data acquired by the sensor section 110. Furthermore,the user situation recognition section 101 may perform recognition of agesture or a voice. Moreover, the user situation recognition section 101is also able to perform tracking of a change in direction, a movement,etc. of a face (head tracking, position tracking).

(Surroundings Situation Recognition Section 102)

The surroundings situation recognition section 102 performs objectrecognition, spatial recognition, etc. in surroundings as a surroundingssituation on the basis of various sensing data acquired by the sensorsection 110.

(Disposition Information Management Section 103)

The disposition information management section 103 performs managementof information about disposition of a real object. For example, thedisposition information management section 103 acquires dispositionresult information of a real object by the user on the basis of varioussensing data acquired by the sensor section 110, and stores thedisposition result information in the storage section 160. Furthermore,the disposition information management section 103 may transmit thedisposition result information accumulated in the storage section 160 tothe server 20.

(Mode Setting Section 104)

The mode setting section 104 performs setting of the preparation mode orthe normal mode as an operation mode of an application. The preparationmode is a mode in which in a case where the setter or someone preparesand disposes a real object with which a virtual object has aninteraction in advance, information about the disposition of the realobject is displayed in augmented reality. The normal mode is a mode inwhich the virtual object that has the interaction with the disposed realobject is displayed; for example, in a case of a game application, aplayer (a general user) is a subject. The mode setting section 104 mayset the operation mode to a mode selected by a user having predeterminedauthority (for example, the setter). It is to be noted that the sameperson may use the preparation mode and the normal mode.

(Display Control Section 105)

The display control section 105 performs augmented reality displaycontrol of a virtual object. Specifically, the display control section105 superimposes and displays a virtual object on a real space inaccordance with the application with use of the display section 120. Atthis time, on the basis of recognition results of the user situationrecognition section 101 and the surroundings situation recognitionsection 102, the display control section 105 may perform display controlto make the virtual object look as if it actually exists in the realspace. Specifically, the display control section 105 changes the virtualobject in accordance with the user's head movement, a change in theposture, the position, etc. (what is called a head tracking function anda position tracking function).

In a case where the preparation mode has been set by the mode settingsection 104, on the basis of information about a real object with whicha virtual object has an interaction, the display control section 105displays, in augmented reality, information that supports disposition ofthe real object. The information about the real object with which thevirtual object has the interaction (hereinafter, also referred to as the“real object information”) is, as described above, inputted by thedeveloper upon defining behavior (the interaction) of the virtualobject. Information about the behavior of the virtual object includesinformation about an interaction point (for example, a relative positionfrom a reference point in a predetermined area). Furthermore, the realobject information includes alternative processing in a case where thereis not the real object in addition to a general name, a shape, amaterial, disposition information, etc. Here, FIG. 4 illustrates anexample of the real object information according to the presentembodiment.

As illustrated in FIG. 4, categories of the real object information areassumed to include, for example, an ID of an associated virtual object,the general name of a real object, disposition information (such as aposition, a direction, etc.), 3D asset (an example of a virtual objectof a real object used in the preparation mode), size information, amaterial, a weight, a color, contents of interaction with a virtualobject, a reference image (an illustration, a photo, etc.) of a realobject, alternative processing in a case where a concerned real objecthas not been disposed, a comment about disposition from a developer,etc.

Disposition information may be, for example, information about arelative position and a direction from a predetermined reference pointin an area. The predetermined reference point may be, for example, theentrance of a room, a window, a predetermined piece of furniture, or thelike. In a case of an event application to be performed in a particularplace, a spot where an interaction is produced (an interaction point)may be set in advance, and thus disposition (a position, a direction,etc.) of an associated virtual object may also be set at the same time.

Examples of alternative processing when a real object has not beendisposed include not displaying a virtual object, not producing a sound,displaying the real object as a virtual object, displaying a virtualobject even if there is no real object, not starting an application (notexecuting an application for the normal mode), etc.

It is to be noted that the real object information illustrated in FIG. 4is an example, and the present embodiment is not limited thereto. Forexample, the real object information may include environment informationabout surroundings of the real object (such as making its surroundingsbright or dark).

Furthermore, in a case where the normal mode has been set by the modesetting section 104, the display control section 105 performs augmentedreality display of a virtual object that has an interaction with thedisposed real object. The application is embedded with an interactionpoint (designation of an area, designation of a place in the area),contents of the interaction, display data of a 3D model or the like ofthe virtual object that has the interaction, sound effect data at thetime of the interaction, preferred real object information in theinteraction point, etc. On the basis of these pieces of information, thedisplay control section 105 performs display control of moving thevirtual object at the designated place (the interaction point). At thistime, the display control section 105 refers to disposition resultinformation acquired during the preparation mode, and determinesbehavior of the virtual object in accordance with a disposition result.That is, on the basis of the disposition result, the display controlsection 105 performs alternative processing as necessary, which allowsan action of the virtual object or a sound effect not to becomeunnatural.

The disposition result information may be stored in the storage section160, or may be acquired from the server 20. The disposition resultinformation is assumed to include not only information about thepresence or absence of disposition of a real object, such as thatwhether or not a designated real object has been disposed as designated,but also information that another real object (a substitute) has beendisposed, such as that “a real object having the same shape but adifferent material has been disposed” or that “a real object having thesame shape and the same material but a smaller size has been disposed”.It is to be noted that it is also possible to recognize disposition of areal object in real time by means of the outward-facing camera 111 andacquire a difference from favorable disposition of a real object set inadvance in an application. However, in a case where a recognitionprocessing load is large, and it is difficult to judge the material andthe weight or accurately grasp the difference from a favorable realobject set in advance, it is more desirable to refer to the dispositionresult information acquired during the preparation mode. Furthermore, ina case of real-time recognition, a state of disposition of a real objectis not known without actually going to a place of the disposition;therefore, also from a point of view that the state of disposition ofthe real object is able to be grasped in advance in a case where anapplication is started in the normal mode, it is preferable to refer tothe disposition result information acquired during the preparation mode.

It is to be noted that in control of an interaction of a virtual object,it is also possible to output a sound effect, a vibration, or the like.At the time of the interaction of the virtual object, the controller 100may perform control of outputting a prescribed sound effect from thespeaker 130. Furthermore, at this time, the controller 100 may refer tothe disposition result information and output a sound effect inaccordance with the material, etc. of an actually disposed real object.

<2-3. Display Section 120>

The display section 120 has a function of performing augmented realitydisplay of a virtual object. For example, the display section 120 isrealized by an optically transmissive display, and uses, for example, ahologram optical technique to perform display. Furthermore, the displaysection 120 may be realized by an LCD (Liquid Crystal Display) device,an OLED (Organic Light Emitting Diode) device, or the like. Moreover,the display section 120 may be of a transmissive type, asemi-transmissive type, or a non-transmissive type. Furthermore, theoptical transmittance of the display section 120 may be controlled bythe controller 100.

<2-4. Speaker 130>

The speaker 130 reproduces an audio signal in accordance with control bythe controller 100.

<2-5. Communication Section 140>

The communication section 140 is a communication module for performingtransmission and reception of data with another device viawired/wireless means. The communication section 140 performs wirelesscommunication directly or through a network access point with anexternal device, for example, by means such as a wired LAN (Local AreaNetwork), a wireless LAN, Wi-Fi (registered trademark) (WirelessFidelity), infrared communication, Bluetooth (registered trademark), andshort-range/non-contact communication.

<2-6. Operation Input Section 150>

The operation input section 150 is realized by an operating memberhaving a physical structure, such as a switch, a button, or a lever.

<2-7. Storage Section 160>

The storage section 160 is realized by a ROM (Read Only Memory) thatstores a program, a calculation parameter, etc. that are used in theabove-described processing by the controller 100 and a RAM (RandomAccess Memory) that temporarily stores a parameter, etc. that change asappropriate.

Furthermore, the storage section 160 stores spatial recognitioninformation (spatial map data), an application, a disposition resultinformation DB (database). The disposition result information DB may bestored in the server 20.

As above, the configuration of the information processing terminal 10according to the present embodiment has been specifically described. Itis to be noted that the configuration of the information processingterminal 10 according to the present embodiment is not limited to theexample illustrated in FIG. 3. For example, the information processingterminal 10 may include a plurality of devices. For example, theinformation processing terminal 10 may have a system configurationincluding an HMD worn by a user and a smartphone. In this case, forexample, the functions that the controller 100 has may be provided tothe smartphone, and information presentation through the HMD may beperformed in accordance with control by the smartphone.

Furthermore, at least a portion of the sensor section 110 may be anexternal sensor (an environment-side sensor, for example, such as acamera set in a room, a depth sensor, a mike, an infrared sensor, or aluminance sensor).

Moreover, at least some of the functions that the controller 100 of theinformation processing terminal 10 has may exist in another device withwhich a communication connection is established through thecommunication section 140. For example, at least some of the functionsthat the controller 100 of the information processing terminal 10 hasmay be provided in an intermediate server or a cloud server (the server20) or the like on the Internet.

3. OPERATION PROCESSING

Subsequently, operation processing of the information processing systemaccording to the present embodiment is specifically described withdrawings.

<3-1. Operation Processing in Preparation Mode>

FIG. 5 is a flowchart illustrating an example of the flow of operationprocessing in the preparation mode according to the present embodiment.As illustrated in FIG. 5, first, the controller 100 of the informationprocessing terminal 10 starts an application (step S103), and checks thesetting of the operation mode (step S106).

Next, in a case of setting to the preparation mode (step S106/thepreparation mode), the display control section 105 acquires real objectinformation embedded in the application (step S109). Alternatively, thedisplay control section 105 acquires the real object information fromthe server 20.

Then, the display control section 105 generates a 3D model for thepreparation mode (which is an example of a virtual object of the realobject used in the preparation mode, and may not necessarily be a 3Dimage) on the basis of the acquired real object information (step S112).Generating display for the preparation mode from the real objectinformation (a real object parameter) in this way makes it possible toreduce man-hours to create an instruction for each application andcreate an application exclusively for preparation.

Next, the controller 100 starts control of guiding a user (a setter) toa disposition place of the real object on the basis of dispositioninformation extracted from the real object information (step S115).Specifically, the controller 100 determines the user's self-location bymeans of the SLAM technique or the like, and guides the user to adesignated disposition place using the display section 120 or thespeaker 130. At this time, the controller 100 may notify the user ofinformation about the real object to be brought to the disposition placebefore the guiding is started. For example, the controller 100 maydisplay a reference image of the real object and also notify a messagesuch as that “Please move to the back of the room with a garbage canlike this”. Furthermore, the controller 100 may display the informationabout the real object to be disposed during the guiding.

Then, in a case where the user has moved to the disposition place inaccordance with the guiding, the display control section 105 displaysthe real object information about the real object to be disposed (stepS118). Specifically, as described with reference to FIG. 1, a virtualobject of the real object (the generated 3D model) is displayed inaugmented reality in the designated disposition place to more clearlyindicate the disposition place of the real object. At this time, thedisplay control section 105 displays detailed information about the realobject as well, or displays, as a test, the virtual object that has aninteraction with the real object, thereby making it possible to furthersupport disposition of the real object. This enables even the setter whodoes not know contents of the application to dispose the real objectnecessary for the application in a position and direction that adeveloper intends.

Next, the disposition information management section 103 acquiresfeedback about the disposition of the real object (in the presentspecification, referred to as disposition result information), andrecords the disposition result information in the disposition resultinformation DB (step S121). The disposition result information DB mayhave been stored in the storage section 160, or may have been stored inthe server 20. In a case where the disposition of the real object isperformed by a plurality of members of staff, the result (thedisposition result information) may be made sharable on a network. Amethod of acquiring the disposition result information is notparticularly limited; however, for example, as illustrated in FIG. 1, ina case where a designated real object has been disposed, the checkbox441 may be checked off. In a case where the disposition has not beenable to be performed, this process may end without checking off, orinformation that the disposition has not been able to be performed maybe inputted. Furthermore, in a case where another real object (asubstitute) has been disposed, such as that “a real object having thesame shape but a different material has been disposed” or that “a realobject having the same shape and the same material but a smaller sizehas been disposed”, the user (the setter) may input information aboutthe substitute and check off the checkbox.

The processes in the steps S115 to S112 described above are repeatedwith respect to all real objects set in the real object information(step S124). It is to be noted that as described above, in a case wherethe disposition is performed by a plurality of members of staff, themembers of the staff may share disposition result information, and areal object that has been disposed by another member of the staff may beremoved from subjects of the guiding as appropriate.

As above, the operation processing in the preparation mode has beenspecifically described.

<3-2. Operation Processing in Normal Mode>

FIG. 6 is a flowchart illustrating an example of the flow of operationprocessing in the normal mode according to the present embodiment. Instep S106 illustrated in FIG. 5, in a case where the normal mode is set(the step S106/the normal mode), as illustrated in FIG. 6, thecontroller 100 analyzes the situation of anindispensable-for-disposition real object from the disposition resultinformation DB (step S123). The indispensable-for-disposition realobject is a real object indispensable for UX (User Experience) of anapplication, and is set by a developer in advance. Information about theindispensable-for-disposition real object is included in, for example,real object information embedded in the application.

Next, the controller 100 determines whether or not the application isable to be executed (step S126). Specifically, in a case where theindispensable-for-disposition real object has not been disposed, thecontroller 100 determines that the application is not able to beexecuted, and, in a case where the indispensable-for-disposition realobject has been disposed, the controller 100 determines that theapplication is able to be executed.

Then, in a case where it has been determined that the application is notable to be executed (step S126/No), the display control section 105presents a user (a player) with a message about the execution being notpossible (step S129). For example, the display control section 105 givesa notification such as that “An issue occurred. Please call staff”. Thismakes it possible to avoid the starting of the application (for example,causing a game to go on) in a situation where theindispensable-for-disposition real object has not been disposed (i.e., asituation where user experience deteriorates).

Meanwhile, in a case where it has been determined that the applicationis able to be executed (step S126/Yes), the display control section 105starts the execution of the application in the normal mode (in a case ofan event application, starts introduction, etc. for the player), andchecks whether or not the player has arrived at a designated interactionpoint (step S132).

Next, in a case where the arrival at the interaction point has beenconfirmed, the controller 100 refers to the disposition resultinformation DB and checks whether or not a designated real object hasbeen disposed in the interaction point (step S135).

Then, in a case where the designated real object has been disposed (stepS135/Yes), the controller 100 performs prescribed normal processing (apredefined interaction of a virtual object) (step S138).

Meanwhile, in a case where the designated real object has not beendisposed (step S135/No), the controller 100 checks an alternativeprocessing flag (i.e., whether or not alternative processing has beenregistered in the real object information) (step S141), and, in a casewhere the alternative processing has been registered, the controller 100performs the alternative processing (step S144). The alternativeprocessing includes alternative processing appropriate for a substitutein a case where the substitute has been disposed in addition toalternative processing in a case where a real object has not beendisposed.

Examples of the alternative processing include not displaying a virtualobject, generating a 3D model of an image of a real object anddisplaying the 3D model as a virtual object, changing an interaction(for example, changing to an interaction unrelated to a real object),etc. Examples of the changing to an interaction unrelated to a realobject include changing, for example, behavior of popping out of thereal object to behavior of popping out of a floor or a wall.

Thus, even in a case where the operating side has not been able todispose the real object unexpectedly, causing a virtual object toperform alternative behavior makes it possible to provide theapplication without making the behavior look unnatural and bringing theuser back to reality.

Such an information processing system according to the presentembodiment is applicable to, for example, an AR game event using areality space as described below, and in a case where a real object hasnot been disposed, it is possible to prevent a feeling of strangeness byperforming given alternative processing.

-   -   An interaction where a virtual object (such as a game character)        collides with a property such as a garbage can in a reality        space, and a sound is produced    -   An interaction where the game character that is a virtual object        gives an explanation about the real space (such as an        explanation about a painting that is a real object)    -   An interaction that makes it appear as if a user is sharing a        scene, such as that the game character that is a virtual object        looks at “a book” or “a poster” (a real object) in the real        space and gets a surprise or starts laughing    -   An interaction where the game character that is a virtual object        runs around and stumbles over a real object    -   In a case where a game is caused to go on by the game character        that is a virtual object guiding a user to a real object and the        user having an interaction with the real object (for example,        such as pushing a button, holding his/her hand over a sensor, or        picking up an item) (the game is not able to go on further        unless a real object is disposed; however, the user's        interaction is produced by alternative processing (for example,        display of a virtual object of a button), which makes it        possible to cause the game to go on)

Of the examples described above, the “interaction where a virtual object(such as a game character) collides with a property such as a garbagecan in a reality space, and a sound is produced” is described withreference to FIG. 7. As illustrated in a left part of FIG. 7, in a casewhere a designated real object 50 has been disposed in a designatedplace, there is presented normal processing, i.e., an interaction wherethe virtual object 40 (such as a game character) collides with the realobject 50, and a sound is produced. Meanwhile, in a case where it hasbeen found by referring to the disposition result information DB thatthe designated real object 50 has not been disposed, if the normalprocessing is performed, the interaction becomes unnatural (a collisionaction and sound is produced out of nowhere) as illustrated in an upperright part of FIG. 7. Accordingly, performing alternative processing asillustrated in a lower right part of FIG. 7 makes it possible to preventa feeling of strangeness. Specifically, displaying a virtual object 46of the real object, eliminating the collision sound, or changing anaction of a virtual object 48 to an action unrelated to the real object,such as behavior of slipping and falling on the spot makes it possibleto prevent the feeling of strangeness in a case where the real objecthas not been disposed.

The processes in the steps S132 to S143 described above are repeated ateach interaction point until the application has been closed (stepS146).

As above, the example of the operation processing according to thepresent embodiment has been described. It is to be noted that theoperation processing illustrated in FIGS. 5 and 6 is an example, and thepresent disclosure is not limited to the example illustrated in FIGS. 5and 6. For example, the present disclosure is not limited to the orderof steps illustrated in FIGS. 5 and 6. At least some of the steps may beperformed in parallel, or may be performed in reverse order. Forexample, the step S112 in FIG. 5 may be performed after the guidingillustrated in the step S115.

Furthermore, all the processes illustrated in FIGS. 5 and 6 may notnecessarily be performed. For example, only a disposition place (such asan area name) may be presented without the guiding illustrated in thestep S115 in FIG. 5.

Moreover, all the processes illustrated in FIGS. 5 and 6 may notnecessarily be performed by a single device. For example, theacquisition of the real object information illustrated in the step S109in FIG. 5, the generation of the 3D model illustrated in the step S112,and the situation analysis of the indispensable-for-disposition realobject illustrated in the step S123 in FIG. 6 may be performed by theserver 20, and the other processes may be performed by the informationprocessing terminal 10.

4. CONCLUSION

As described above, in the information processing system according tothe embodiment of the present disclosure, it is possible to supportdisposition of a real object related to a virtual object and preventdeterioration of user experience in a case where the virtual object isdisplayed.

A preferred embodiment(s) of the present disclosure has/have beendescribed above in detail with reference to the accompanying drawings,but the technical scope of the present disclosure is not limited to suchan embodiment(s). A person skilled in the art may find variousalterations and modifications within the scope of the appended claims,and it should be understood that they will naturally come under thetechnical scope of the present disclosure.

For example, it is also possible to create a computer program causinghardware such as the CPU, the ROM, and the RAM built into theabove-described information processing terminal 10 or the server 20 tofulfill the functions of the information processing terminal 10 or theserver 20. Furthermore, there is also provided a computer-readablestorage medium in which the computer program is stored.

In addition, the effects described herein are merely illustrative andexemplary, and not limitative. That is, the technology according to thepresent disclosure may exert other effects that are apparent to thoseskilled in the art from the description herein, in addition to theeffects described above or in place of the effects described above.

It is to be noted that the present technology may take configurations asfollow.

(1)

An information processing apparatus including a controller thatperforms: control of displaying information in augmented reality in apreparation mode in which a real object is disposed, the informationthat supports disposition of the real object on the basis of prerecordedinformation about the real object with which a virtual object has aninteraction; and control of recording disposition result information ofthe real object in a disposition result information database referred toin a normal mode in which the virtual object that has the interactionwith the disposed real object is displayed.

(2)

The information processing apparatus according to (1), in which thecontroller displays information that indicates the disposition of thereal object as the information that supports the disposition of the realobject.

(3)

The information processing apparatus according to (1) or (2), in whichthe controller displays, in augmented reality, a virtual object of thereal object in a designated disposition place in a real space as theinformation that supports the disposition of the real object.

(4)

The information processing apparatus according to (1) or (2), in whichthe controller displays, as the information that supports thedisposition of the real object, a virtual object of the real object andthe virtual object that has the interaction with the real object in adesignated disposition place in a real space.

(5)

The information processing apparatus according to any one of (1) to (4),in which the controller performs display for guiding a user to adisposition place of the real object as the information that supportsthe disposition of the real object.

(6)

The information processing apparatus according to (5), in which thecontroller gives a notification that prompts the user to prepare thereal object to be set in the disposition place upon starting theguiding.

(7)

The information processing apparatus according to any one of (1) to (6),in which the controller displays detailed information about the realobject as the information that supports the disposition of the realobject.

(8)

The information processing apparatus according to (7), in which thedetailed information about the real object includes at least any one ofa name of the real object, a 3D asset, size information, materialinformation, weight information, interaction contents, a captured image,an illustration image, or a comment.

(9)

The information processing apparatus according to any one of (1) to (8),in which the information about the real object is stored in associationwith the virtual object.

(10)

The information processing apparatus according to any one of (1) to (9),in which in the normal mode, the controller performs display control ofthe virtual object in a designated place set in accordance with thedisposition result information of the real object stored in thedisposition result information database.

(11)

The information processing apparatus according to (10), in which in acase where it is determined from the disposition result information ofthe real object that the real object specified has not been disposed inthe preparation mode, the controller stops execution of the normal mode.

(12)

The information processing apparatus according to (10) or (11), in whichin a case where it is confirmed from the disposition result informationof the real object that the real object designated has not been disposedin an interaction point, the controller performs designated alternativeprocessing for display of the virtual object.

(13)

The information processing apparatus according to (12), in which as thealternative processing, the controller stops display of the virtualobject, or changes to an interaction unrelated to the real object.

(14)

The information processing apparatus according to (12), in which as thealternative processing, the controller displays a virtual object of thereal object in a designated disposition place, and executes a setinteraction of the virtual object.

(15)

The information processing apparatus according to (10) or (11), in whichin a case where it is confirmed from the disposition result informationof the real object that a real object different from a designated realobject has been disposed in an interaction point, the controller changesto an interaction appropriate for the different real object.

(16)

The information processing apparatus according to (15), in which thecontroller changes an interaction of the virtual object in accordancewith at least either material information or size information of thedifferent real object.

(17)

The information processing apparatus according to (16), in which thechanging of the interaction of the virtual object includes changing ofan action of the virtual object or changing of a sound.

(18)

An information processing method that is performed by a processor, theinformation processing method including:

displaying information in augmented reality in a preparation mode inwhich a real object is disposed, the information that supportsdisposition of the real object on the basis of prerecorded informationabout the real object with which a virtual object has an interaction;and

recording disposition result information of the real object in adisposition result information database referred to in a normal mode inwhich the virtual object that has the interaction with the disposed realobject is displayed.

(19)

A program causing a computer to function as a controller that performs:

when in preparation mode in which a real object is disposed, control ofdisplaying, in augmented reality, information that supports dispositionof the real object on the basis of prerecorded information about thereal object with which a virtual object has an interaction; and

control of recording disposition result information of the real objectin a disposition result information database referred to when in normalmode in which the virtual object that has the interaction with thedisposed real object is displayed.

REFERENCE SIGNS LIST

-   10 information processing terminal-   100 controller-   101 user situation recognition section-   102 surroundings situation recognition section-   103 disposition information management section-   104 mode setting section-   105 display control section-   110 sensor section-   111 outward-facing camera-   112 inward-facing camera-   113 mike-   114 gyro sensor-   115 acceleration sensor-   116 direction sensor-   117 position measurement section-   120 display section-   130 speaker-   140 communication section-   150 operation input section-   160 storage section-   20 server

1. An information processing apparatus comprising a controller thatperforms: control of displaying information in augmented reality in apreparation mode in which a real object is disposed, the informationthat supports disposition of the real object on a basis of prerecordedinformation about the real object with which a virtual object has aninteraction; and control of recording disposition result information ofthe real object in a disposition result information database referred toin a normal mode in which the virtual object that has the interactionwith the disposed real object is displayed.
 2. The informationprocessing apparatus according to claim 1, wherein the controllerdisplays information that indicates the disposition of the real objectas the information that supports the disposition of the real object. 3.The information processing apparatus according to claim 1, wherein thecontroller displays, in augmented reality, a virtual object of the realobject in a designated disposition place in a real space as theinformation that supports the disposition of the real object.
 4. Theinformation processing apparatus according to claim 1, wherein thecontroller displays, as the information that supports the disposition ofthe real object, a virtual object of the real object and the virtualobject that has the interaction with the real object in a designateddisposition place in a real space.
 5. The information processingapparatus according to claim 1, wherein the controller performs displayfor guiding a user to a disposition place of the real object as theinformation that supports the disposition of the real object.
 6. Theinformation processing apparatus according to claim 5, wherein thecontroller gives a notification that prompts the user to prepare thereal object to be set in the disposition place upon starting theguiding.
 7. The information processing apparatus according to claim 1,wherein the controller displays detailed information about the realobject as the information that supports the disposition of the realobject.
 8. The information processing apparatus according to claim 7,wherein the detailed information about the real object includes at leastany one of a name of the real object, a 3D asset, size information,material information, weight information, interaction contents, acaptured image, an illustration image, or a comment.
 9. The informationprocessing apparatus according to claim 1, wherein the information aboutthe real object is stored in association with the virtual object. 10.The information processing apparatus according to claim 1, wherein inthe normal mode, the controller performs display control of the virtualobject in a designated place set in accordance with the dispositionresult information of the real object stored in the disposition resultinformation database.
 11. The information processing apparatus accordingto claim 10, wherein in a case where it is determined from thedisposition result information of the real object that the real objectspecified has not been disposed in the preparation mode, the controllerstops execution of the normal mode.
 12. The information processingapparatus according to claim 10, wherein in a case where it is confirmedfrom the disposition result information of the real object that the realobject designated has not been disposed in an interaction point, thecontroller performs designated alternative processing for display of thevirtual object.
 13. The information processing apparatus according toclaim 12, wherein as the alternative processing, the controller stopsdisplay of the virtual object, or changes to an interaction unrelated tothe real object.
 14. The information processing apparatus according toclaim 12, wherein as the alternative processing, the controller displaysa virtual object of the real object in a designated disposition place,and executes a set interaction of the virtual object.
 15. Theinformation processing apparatus according to claim 10, wherein in acase where it is confirmed from the disposition result information ofthe real object that a real object different from a designated realobject has been disposed in an interaction point, the controller changesto an interaction appropriate for the different real object.
 16. Theinformation processing apparatus according to claim 15, wherein thecontroller changes an interaction of the virtual object in accordancewith at least either material information or size information of thedifferent real object.
 17. The information processing apparatusaccording to claim 16, wherein the changing of the interaction of thevirtual object includes changing of an action of the virtual object orchanging of a sound.
 18. An information processing method that isperformed by a processor, the information processing method comprising:displaying information in augmented reality in a preparation mode inwhich a real object is disposed, the information that supportsdisposition of the real object on a basis of prerecorded informationabout the real object with which a virtual object has an interaction;and recording disposition result information of the real object in adisposition result information database referred to in a normal mode inwhich the virtual object that has the interaction with the disposed realobject is displayed.
 19. A program causing a computer to function as acontroller that performs: control of displaying information in augmentedreality in a preparation mode in which a real object is disposed, theinformation that supports disposition of the real object on a basis ofprerecorded information about the real object with which a virtualobject has an interaction; and control of recording disposition resultinformation of the real object in a disposition result informationdatabase referred to in a normal mode in which the virtual object thathas the interaction with the disposed real object is displayed.